{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先配置相关的环境，Linux系统，更换软件源。安装Miniconda等等。之后利用conda命令安装常用的库包括（numpy,pandas,sklearn,tensorflow,pytorch,opencv)注意在安装库时，应该选择与本电脑Python版本相对应的版本进行安装（一般不要尝试最新的一版）。\n",
    "在安装过程中学会更换软件源，建立编辑文本等常用操作命令。同时，学会用命令行安装软件并建立快捷方式的操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 预处理工作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入函数库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入数据集\n",
    "机器学习中往往会需要利用数据集对算法进行训练，因此学习导入数据集是第一步\n",
    "数据集通常是.csv格式。CSV文件以文本形式保存表格数据。文件的每一行是一条数据记录。我们使用Pandas的read_csv方法读取本地csv文件为一个数据帧。然后，从数据帧中制作自变量和因变量的矩阵和向量。具体的步骤如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['France' 44.0 72000.0]\n",
      " ['Spain' 27.0 48000.0]\n",
      " ['Germany' 30.0 54000.0]\n",
      " ['Spain' 38.0 61000.0]\n",
      " ['Germany' 40.0 nan]\n",
      " ['France' 35.0 58000.0]\n",
      " ['Spain' nan 52000.0]\n",
      " ['France' 48.0 79000.0]\n",
      " ['Germany' 50.0 83000.0]\n",
      " ['France' 37.0 67000.0]]\n",
      "['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']\n"
     ]
    }
   ],
   "source": [
    "dataset = pd.read_csv('../datasets/Data.csv')\n",
    "#不包括最后一列的所有列（python切片）\n",
    "X = dataset.iloc[ : , :-1].values;\n",
    "#最后一列\n",
    "Y = dataset.iloc[ : , 3].values;\n",
    "print(X)\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理丢失数据\n",
    "在获取数据的过程中，经常会遇到数据丢失的情况，为了减小丢失数据对于算法的影响，一般我们对我们可以用整列的平均值或中间值替换丢失的数据。我们用sklearn.preprocessing库中的Imputer类完成这项任务："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['France' 44.0 72000.0]\n",
      " ['Spain' 27.0 48000.0]\n",
      " ['Germany' 30.0 54000.0]\n",
      " ['Spain' 38.0 61000.0]\n",
      " ['Germany' 40.0 63777.77777777778]\n",
      " ['France' 35.0 58000.0]\n",
      " ['Spain' 38.77777777777778 52000.0]\n",
      " ['France' 48.0 79000.0]\n",
      " ['Germany' 50.0 83000.0]\n",
      " ['France' 37.0 67000.0]]\n"
     ]
    }
   ],
   "source": [
    "#sklearn版本不同，操作也不同，使用时需要根据自己的版本查阅Imputer和SimpleImputer的操作 \n",
    "#对比数据发现，原来为nan的值现在被均值填充其他的值不变\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(missing_values = np.nan, strategy = \"mean\")\n",
    "imputer = imputer.fit(X[ : , 1:3])\n",
    "X[ : , 1:3] = imputer.transform(X[ : , 1:3])\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 解析分类数据\n",
    "因为很多样本的标签通常为单词而并非数据因此我们在开始前还需要进行解析分类数据的工作，从sklearn.preprocessing库导入LabelEncoder类实现这一功能："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LabelEncoder,OnehotEncoder功能解释:\n",
    "LabelEncoder是统计出现的数据，并打上标签，例如在dataset中，一共有三类'france''spain''germany'所以定义出来的向量为[0,1,2]。\n",
    "OnehotEncoder叫做独热编码，因为为了避免在训练中标签大小对于训练结果产生影响，统计总共出现的数据量并生成相应地向量，保证每个标签之间的距离相等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X former\n",
      "[[1 0.0 0.0 44.0 72000.0]\n",
      " [0 0.0 1.0 27.0 48000.0]\n",
      " [0 1.0 0.0 30.0 54000.0]\n",
      " [0 0.0 1.0 38.0 61000.0]\n",
      " [0 1.0 0.0 40.0 63777.77777777778]\n",
      " [1 0.0 0.0 35.0 58000.0]\n",
      " [0 0.0 1.0 38.77777777777778 52000.0]\n",
      " [1 0.0 0.0 48.0 79000.0]\n",
      " [0 1.0 0.0 50.0 83000.0]\n",
      " [1 0.0 0.0 37.0 67000.0]]\n",
      "X\n",
      "[[0.0 1.0 0.0 0.0 44.0 72000.0]\n",
      " [1.0 0.0 0.0 1.0 27.0 48000.0]\n",
      " [1.0 0.0 1.0 0.0 30.0 54000.0]\n",
      " [1.0 0.0 0.0 1.0 38.0 61000.0]\n",
      " [1.0 0.0 1.0 0.0 40.0 63777.77777777778]\n",
      " [0.0 1.0 0.0 0.0 35.0 58000.0]\n",
      " [1.0 0.0 0.0 1.0 38.77777777777778 52000.0]\n",
      " [0.0 1.0 0.0 0.0 48.0 79000.0]\n",
      " [1.0 0.0 1.0 0.0 50.0 83000.0]\n",
      " [0.0 1.0 0.0 0.0 37.0 67000.0]]\n",
      "Y\n",
      "[0 1 0 0 1 1 0 1 0 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
    "labelencoder_X = LabelEncoder()\n",
    "X[ : , 0] = labelencoder_X.fit_transform(X[ : , 0])\n",
    "print(\"X former\")\n",
    "print(X)                \n",
    "#在运行时，为了比较两种函数的区别，先把下面的程序注释掉\n",
    "onehotencoder = ColumnTransformer([(\"country\",OneHotEncoder(),[0])],remainder = 'passthrough')\n",
    "X = onehotencoder.fit_transform(X)#toarray\n",
    "labelencoder_Y = LabelEncoder()\n",
    "Y =  labelencoder_Y.fit_transform(Y)\n",
    "print(\"X\")\n",
    "print(X)\n",
    "print(\"Y\")\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 拆分数据\n",
    "为了训练与验证的需要，我们往往需要把数据集拆分为训练数据和测试数据，两者比例一般是80:20。我们导入sklearn.model_selection库中的train_test_split()方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------\n",
      "Step 5: Splitting the datasets into training sets and Test sets\n",
      "X_train\n",
      "[[0.0 1.0 0.0 40.0 63777.77777777778]\n",
      " [1.0 0.0 0.0 37.0 67000.0]\n",
      " [0.0 0.0 1.0 27.0 48000.0]\n",
      " [0.0 0.0 1.0 38.77777777777778 52000.0]\n",
      " [1.0 0.0 0.0 48.0 79000.0]\n",
      " [0.0 0.0 1.0 38.0 61000.0]\n",
      " [1.0 0.0 0.0 44.0 72000.0]\n",
      " [1.0 0.0 0.0 35.0 58000.0]]\n",
      "X_test\n",
      "[[0.0 1.0 0.0 30.0 54000.0]\n",
      " [0.0 1.0 0.0 50.0 83000.0]]\n",
      "Y_train\n",
      "[1 1 1 0 1 0 0 1]\n",
      "Y_test\n",
      "[0 0]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split( X , Y , test_size = 0.2, random_state = 0)\n",
    "print(\"---------------------\")\n",
    "print(\"Step 5: Splitting the datasets into training sets and Test sets\")\n",
    "print(\"X_train\")\n",
    "print(X_train)\n",
    "print(\"X_test\")\n",
    "print(X_test)\n",
    "print(\"Y_train\")\n",
    "print(Y_train)\n",
    "print(\"Y_test\")\n",
    "print(Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 特征量化\n",
    "因为训练时参数幅度的不同可能会影响到实际的训练结果，为了保证每个因素都被平等地考虑，因此我们可以采用归一化处理的方式（大部分模型算法使用两点间的欧氏距离表示，但此特征在幅度、单位和范围姿态问题上变化很大。在距离计算中，高幅度的特征比低幅度特征权重更大。可用特征标准化或Z值归一化解决。导入sklearn.preprocessing库的StandardScalar类。）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------\n",
      "Step 6: Feature Scaling\n",
      "X_train\n",
      "[[-1.          2.64575131 -0.77459667  0.26306757  0.12381479]\n",
      " [ 1.         -0.37796447 -0.77459667 -0.25350148  0.46175632]\n",
      " [-1.         -0.37796447  1.29099445 -1.97539832 -1.53093341]\n",
      " [-1.         -0.37796447  1.29099445  0.05261351 -1.11141978]\n",
      " [ 1.         -0.37796447 -0.77459667  1.64058505  1.7202972 ]\n",
      " [-1.         -0.37796447  1.29099445 -0.0813118  -0.16751412]\n",
      " [ 1.         -0.37796447 -0.77459667  0.95182631  0.98614835]\n",
      " [ 1.         -0.37796447 -0.77459667 -0.59788085 -0.48214934]]\n",
      "X_test\n",
      "[[-1.          2.64575131 -0.77459667 -1.45882927 -0.90166297]\n",
      " [-1.          2.64575131 -0.77459667  1.98496442  2.13981082]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "sc_X = StandardScaler()\n",
    "X_train = sc_X.fit_transform(X_train)\n",
    "X_test = sc_X.transform(X_test)\n",
    "print(\"---------------------\")\n",
    "print(\"Step 6: Feature Scaling\")\n",
    "print(\"X_train\")\n",
    "print(X_train)\n",
    "print(\"X_test\")\n",
    "print(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
